The detection of outliers is a task in data mining and machine learning and it’s an important step in data preprocessing. In this paper, in order to detect proximity-based outliers, a non-parametric method is proposed called NPOD. The proposed method is a combination of More
The detection of outliers is a task in data mining and machine learning and it’s an important step in data preprocessing. In this paper, in order to detect proximity-based outliers, a non-parametric method is proposed called NPOD. The proposed method is a combination of distance-based and density-based methods and has the ability to detect outliers in both local and global scenarios. This method does not require to determine any parameters of neighborhood radius, the threshold of existing points in the neighborhood radius, and the nearest neighbor parameters. In order to detect outliers, a new method of scoring is presented. Experimental results on the UCI datasets show that this algorithm, in spite of being non-parametric, has comparable results with previous methods. Also in some cases, it has the best performance.
Manuscript profile
One of the challenges of high dimensional outlier detection problem is the curse of dimensionality which irrelevant dimensions (features) lead to hidden outliers. To solve this problem, some dimensions that contain valuable information to detect outliers are searched to More
One of the challenges of high dimensional outlier detection problem is the curse of dimensionality which irrelevant dimensions (features) lead to hidden outliers. To solve this problem, some dimensions that contain valuable information to detect outliers are searched to make outliers more prominent and detectable by mapping the dataset into the subspace which is constituted of these relevant dimensions/features. This paper proposes an outlier detection method in high dimensional data by introducing a new locally relevant subspace selection and developing a local density-based outlier scoring. First, we present a locally relevant subspace selection method based on local entropy to select a relevant subspace for each data point due to its neighbors. Then, each data point is scored in its relevant subspace using a density-based local outlier scoring method. Our adaptive-bandwidth kernel density estimation method eliminates the slight difference between the density of a normal data point and its neighbors. Thus, normal data are not wrongly detected as outliers. At the same time, our method underestimates the actual density of outlier data points to make them more prominent. The experimental results on several real datasets show that our local entropy-based subspace selection algorithm and the proposed outlier scoring can achieve a high accuracy detection rate for the outlier data.
Manuscript profile
